Test Selection

The book has now been published and the content of this chapter has likely changed substanstially.

How does the Test Runner know what tests to run?

The Test Automation Framework selects the Test Methods to be run at runtime based on attributes of the tests.

Sketch Test Selection embedded from Test Selection.gif

Given that we have written a number of Test Methods (page X) on one or more Testcase Classes (page X), we need to give the Test Runner (page X) some way to find the tests.

Test Selection is a way to pick subsets of tests dynamically.

How It Works

The test automater specifies the subset of tests to be run when invoking the Test Runner by providing test selection criteria. The selection criteria may be based on implicit or explicit attributes of the Testcase Classes or Test Methods.

When To Use It

We should use Test Selection when we wish to run a subset of tests chosen from other test suites and we do not want to maintain a separate structure built using Test Enumeration (page X). A Smoke Test[SCM] suite is a common usage; see Named Test Suites (page X) for other uses.

Implementation Notes

Test Selection can be implemented either by creating a Subset Suite (see Named Test Suite) from an existing Test Suite Object (page X) or we can skip some of the tests within the Test Suite Object as we execute the Testcase Objects (page X) it contains.

As with Test Discovery (page X) and Test Enumeration, Test Selection can be applied at two different levels: selecting Testcase Classes or selecting Test Methods. Test Selection can be built into the Test Automation Framework (page X) or it can be implemented more crudely as part of the build task.

Variation: Testcase Class Selection

We can select the Testcase Classes to be examined for Test Methods several ways. The crudest way to do Testcase Class Selection is simply to place the Testcase Classes into test packages based on some criteria. Unfortunately, this only works for a single test classification scheme and is likely to reduce the value of Tests as Documentation (see Goals of Test Automation on page X). A somewhat more flexible way to do it is to use a naming convention such as "contains 'WebServer'" to select only those classes that verify the behavior of certain parts of the system. This, too, is somewhat constrained it its utility.

The most flexible way to implement Test Selection is within the Test Automation Framework. We can use class attributes (.Net) or annotations (Java) to indicate characteristics of the Testcase Class. The same technique can also be applied at the Test Method level.

Variation: Test Method Selection

When implemented as part of the Test Automation Framework, Test Method Selection can be done by specifying which 'category' (or several) that a Test Method belongs to. This usually requires language support for method attributes (.Net) or annotations (Java). It could also be based on a method name scheme but this is not as flexible and would require tighter coupling to the Test Runner.

Example: Testcase Class Selection using Class Attribute

The following example of Testcase Class Selection is from NUnit. The class attributeCategory("FastSuite") indicates that all the tests in this Testcase Class should be included (or excluded) when the category "FastSuite" is specified in the Test Runner.

[TestFixture]
[Category("FastSuite")]
public class CategorizedTests
{
   [Test]
   public void testFlightConstructor_OK()
   // Methods omitted
}
Example CategorizedTestFixture embedded from CSharp/NUnitExamples/CategorizedTests.cs

Example: Test Method Selection using Method Attribute

This example of Test Method Selection is from NUnit. The method attributeCategory("SmokeTest") indicates that this Test Method should be included (or excluded) when the category "SmokeTest" is specified in the Test Runner.

   [Test]
   [Category("SmokeTests")]
   public void testFlightMileage_asKm()
   {
      // setup fixture
      Flight newFlight = new Flight(validFlightNumber);
      newFlight.setMileage(1122);
      // exercise mileage translater
      int actualKilometres = newFlight.getMileageAsKm();   
      int expectedKilometres = 1810;
      // verify results
      Assert.AreEqual( expectedKilometres, actualKilometres);
   }
Example CategorizedTestMethod embedded from CSharp/NUnitExamples/CategorizedTests.cs

Page generated at Wed Feb 09 16:39:47 +1100 2011